Context-related arrangements

ABSTRACT

In one arrangement, a first device presents a display that is based on context data, derived from one or more of its sensors. This display is imaged by a camera in a second device. The second device uses context data from its own sensors to assess the information in the captured imagery, and makes a determination about the first device. In another arrangement, social network friend requests are automatically issued, or accepted, based on contextual similarity. In yet another arrangement, delivery of a message is triggered by a contextual circumstance other than (or in addition to) location. In still another arrangement, two or more devices automatically establish an ad hoc network (e.g., Bluetooth pairing) based on contextual parallels. In still another arrangement, historical context information is archived and used in transactions with other devices, e.g., in challenge-response authentication. A great number of other features and arrangements—many involving head-mounted displays—are also detailed.

RELATED APPLICATION DATA

This application claims priority benefit to provisional application61/546,494, filed Oct. 12, 2011.

BACKGROUND AND INTRODUCTION

Context may be defined as any information useful in characterizing thesituation of an entity. An entity is a person, place or object that isconsidered relevant to the interaction between a user and anapplication, including the user and applications themselves.

Such information can be of many sorts, including computing context(network connectivity, memory availability, processor type, CPUcontention, etc.), user context (user profile, location, actions,preferences, nearby friends, social network(s) and situation, etc.),physical context (e.g., lighting, noise level, traffic, etc.), temporalcontext (time of day, day, month, season, etc.), history of the above,etc.

Context information finds many applications in mobile devices. Forexample, “Bump” is a smartphone app for exchanging business card-likeinformation between two users' devices. Each phone running the appsenses a physical “bump” that occurs when two phones are touchedtogether. Each phone sends a time-stamped report of such event to a Bumpserver, together with information about the phone's GPS location, andinformation about the strength of the bump. The Bump server examines theincoming reports to identify corresponding pairs of bumps, based onsimilarity in time, location, and bump strength. When such a match isfound, the server provides to each phone the contact information for theother user. (The same technology can also be used to exchange songs,photos, and other information between bumped devices.) The Bumptechnology is further detailed in patent application 20110191823.Related technology is described in patent application 20110076942.

Location information is another type of context data, and is sometimesused as a fraud-prevention measure, e.g., in connection with credentialpresentation. For example, U.S. Pat. No. 7,503,488 teaches that if adriver's license is presented as a form of identification by a personcashing a check in New York, and the same license is presented an hourlater by a traveler checking in for a flight in Los Angeles, somethingis amiss. Similarly, if a user presents a bankcard to an ATM in Phoenix,while GPS tracking indicates the user's cell phone is in Atlanta, thebank may treat the ATM transaction as suspect.

In accordance with aspects of the present technology, contextinformation is used in novel and useful ways. In one particularimplementation, a first smartphone presents a screen display that isbased on context data, derived from one or more of its sensors. Thisscreen display is imaged by a camera in a second smartphone. The secondsmartphone uses its own context data to assess the information in thecaptured imagery, to make a determination about the first smartphone.

In accordance with another aspect of the present technology, socialnetwork friend requests are automatically issued, or accepted, based oncontextual similarity.

In accordance with still another aspect of the present technology,delivery of a message to a user is triggered by a contextualcircumstance other than (or in addition to) location.

In accordance with yet another aspect of the present technology, two ormore devices automatically establish an ad hoc network (e.g., Bluetoothpairing) based on contextual parallels.

In accordance with still another aspect of the technology, contextinformation is stored in a memory, and serves as a history archive thatis used in transactions with other devices, e.g., in challenge-responseauthentication.

Many other aspects of the technology concern wearable computers (e.g.,head-mounted display systems), and related uses of context.

The foregoing and additional features and advantages of the presenttechnology will be more readily apparent from the following detaileddescription, which proceeds with reference to the accompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram of a representative smartphone.

FIG. 2 is a flow chart showing one particular method of authenticatingone smartphone to another.

FIG. 3 is a flow chart showing one particular method of authenticatingtwo smartphones to each other.

FIG. 4 is a flow chart showing another particular method ofauthenticating one smartphone to another.

FIG. 5 is a flow chart showing one particular method of securelyproviding an access credential from a first smartphone to a secondsmartphone.

FIG. 6 depicts a wearable computer system.

DETAILED DESCRIPTION

FIG. 1 depicts an illustrative smartphone. This phone includes aprocessor and memory. The memory stores various software and informationused by the phone, including operating system software, applicationsoftware, data, etc. (The functionality detailed herein may beimplemented in the operating system of a smartphone, or as applicationsoftware, or as a hybrid.)

The detailed phone further includes a display screen that also serves asa touch input device. It also includes one or more physical userinterface (UI) elements, such as buttons or switches to controldifferent aspects of operation.

The depicted phone also includes various communications technologies. Inaddition to a cellular radio transceiver (e.g., 4G “WiMax” or LTE), thephone also includes WiFi (802.11) and Bluetooth transceivers. Alsoincluded is an RFID interface, which can both transmit signals to RFIDchips (aka Near Field Chips), and also receive responsive signals inreturn.

The detailed smartphone further includes a variety of sensors, includinga 3D accelerometer, a 3D gyroscopic sensor, a 3D magnetometer, abarometric pressure sensor, one or more microphones, one or morecameras, a location sensor (e.g., a GPS module), etc.

The phone also includes various “logical sensors.” These sensors includesoftware modules that take input data from one or (typically) two ormore physical sensors, and draw inferences or otherwise further processthe physical sensor data, to yield enhanced information. (Many logicalsensors retain state. For example, a classifier (a type of logicalsensor) typically has been trained, and relies on that knowledge toperform its action. In contrast, physical sensors are generallystate-less.)

As indicated, one type of logical sensor is a classifier. As is familiarto artisans, classification is the problem of identifying to which of aset of categories (sub-populations) a new observation belongs. (Afamiliar classification problem is identifying email as spam ornot-spam.) An algorithm or procedure that implements classification isknown as a classifier.

Typically, the various classification categories are defined so as tohave immediate meaning to humans (e.g., speech/music/silence,walking/running/flying, etc.), rather than being characterized bynumeric values or ranges of technical parameters divorced from any humanmeaning.

(Classification is often performed by reference to a training set ofdata, comprising observations whose category membership is known.Classification, in this sense, may be regarded as an instance ofsupervised machine learning, i.e., learning in which a training set ofcorrectly-identified observations is available. The correspondingunsupervised procedure is sometimes termed clustering, and involvesgrouping data into categories based on some measure of inherentsimilarity (e.g. the distance between instances, considered as vectorsin a multi-dimensional vector space). For purposes of the presentapplication, classification includes clustering—provided the output ofthe process identifies a category that has semantic meaning to humanusers.)

An exemplary classifier is an audio classifier. An audio classifiertakes audio sensed by a microphone, and classifies it into one ofseveral categories, e.g., speech, music or silence. It may furtherclassify it by volume (for speech and music), e.g., quiet, mid, or loud(based on stored threshold information). In some implementations theaudio classifier may perform a speech-to-text operation, yielding textdata corresponding to speech sampled by the smartphone microphone.

Another logical sensor is a visual classifier. This classifier takesinput imagery sensed by a camera, and again classifies it into one ofseveral categories, e.g., contains-face, contains-nature, etc. Myriadclassifiers are known; a familiar one identifies specific faces in acollection of Picasa or Facebook images. Object classifiers are widelyused in machine vision systems.

Relevant to some embodiments of the present technology are facialexpression classifiers. Such arrangements sense facial features toevaluate the subject's facial affect. From the resulting information,the subject's mental state can be inferred. (Particular examples aredetailed in Affectiva's patent publication US20120243751. Arrangementsfor sensing the user's own facial affect are detailed in Scheirer, etal, Expression Glasses—A Wearable Device for Facial ExpressionRecognition, MIT Media Laboratory Perceptual Computing Section TechnicalReport No. 484, 1999.)

Another logical sensor is an activity classifier. This classifierexamines available sensor data, and makes a conclusion about the user'sactivity. For example, it may take information from a GPS module anddiscern the user's speed and direction (e.g., west at 400 mph), and alsotake information from an audio sensor or classifier that indicates theambient environment has a constant dull roar (indicative of an airplanecabin), and also take information from barometric sensor indicating thesmartphone is at extraordinarily low atmospheric pressure (e.g., 23.5inches of mercury). Based on this data (and optionally historicalinformation about the user), the activity classifier may output dataconcluding that the user is traveling on an airplane—most likely goingfrom New York to Los Angeles.

Many of the examples that follow assume a transaction is about to beperformed between two smartphones. However, the technology can beemployed in many other situations.

In one arrangement, a first smartphone obtains one or more contextparameters from one or more of its sensors (physical or logical). Theseparameters may comprise, by way of example and not limitation: dateand/or time of day, information based on audio recently sampled by thesmartphone, information based on visual stimulus recently sensed by thesmartphone, sensed radio transmissions, sensed RFID data, orientationsensor data, sensed magnetometer data, sensed accelerometer data,barometric sensor data, audio/visual/activity classification data, etc.

Based on such parameter(s), the first smartphone generates an image, andpresents it on the smartphone screen, where it can be sensed by a cameraof a second smartphone (e.g., to authenticate a smartphone transaction).

FIG. 2 illustrates a particular embodiment of such a method. In thisexample, the first smartphone uses the context parameter of phoneorientation (which may be expressed as three values, corresponding tothe three axes of the smartphone). These three data are digitallywatermarked (steganographically encoded) into a “cover image,” such as afacial portrait of the user of the first smartphone. The watermarkedimage is then displayed on the screen of the first smartphone.

The user of a second smartphone positions the second smartphone parallelto the first smartphone, and captures an image of the portrait displayedon the first smartphone screen.

Being roughly parallel, the orientation sensors in the second phoneshould indicate an orientation that corresponds to data output by theorientation sensors in the first phone. E.g., if the first phone is heldso that it “points” north, and is screen-up, with its right edge tippeddown 15 degrees, the second phone should report correspondingorientation parameters.

(“Correspondence” between parameters depends on the configuration of thesecond phone. If the second phone is imaging the first using a cameralens on the “back” side of the second phone, the parameters shouldmatch. However, if the second phone is using a camera lens on the“front,” next to the second phone's screen, the two phones then havereciprocal poses. That is, while the tops of the phones are both pointednorth, the second phone is screen-down (vs. face-up), and its left edgetips down 15 degrees (vs. right edge tips down 15 degrees).)

The second phone decodes the digital watermark that was earlier encodedinto the screen display by the first smartphone. By so-doing, the secondsmartphone extracts the orientation information reported by the firstsmartphone. The second smartphone then checks that the orientationinformation extracted from the watermark encoded by the first phonecorresponds in an expected manner with the orientation information thatits own sensors report (within some threshold range of tolerance). Ifthe two sets of data correspond, this tends to confirm that the secondsmartphone is dealing with a first smartphone that is truthfullyreporting its present context. This is some evidence that the firstsmartphone is trustworthy and is not, e.g., simply replaying recordedcontext information.

If, in contrast, the orientation information that the second smartphoneextracts from the first phone's digital watermark does not correctlycorrespond to the orientation information that its own sensors report,this is regarded as a failure by the first phone to authenticate itself.

While the just-discussed arrangement was based on three contextparameters (i.e., orientation in three dimensions), the same process canbe performed with one or two context parameters, or more than threeparameters.

FIG. 3 details a process similar to that just-discussed, but detailingtwo-way authentication—in which the each smartphone tests the other.

In this arrangement, the first smartphone uses information from alogical sensor: an audio classifier. This audio classification data isencrypted using a private key of the first smartphone. The resultinginformation is encoded in 2D barcode form, and presented on the screenof the first smartphone.

The second smartphone polls its own audio classifier to characterize theambient audio environment. Before, or after, it also captures imagery ofthe barcode presented on the screen of the first smartphone, and decodesthis barcode to retrieve the encrypted data thereby conveyed. The secondsmartphone applies the public key of the first smartphone to retrievethe audio classification data reported by the first smartphone.

The second smartphone then compares the two audio classification datafor correspondence. If they don't match within a suitable errortolerance, authentication of the first smartphone has failed.

Assuming the two reports of audio classification data agree, the secondphone then repeats the process—now authenticating itself to the firstsmartphone. In this phase of operation it senses its own orientation(e.g., in three dimensions, as described above). The second phoneencrypts this orientation information with its own private key, andencodes the result in a barcode, and presents it on the secondsmartphone screen.

The first smartphone engages in a parallel operation. The firstsmartphone is positioned parallel to the second phone and captures animage from its screen display. The 2D barcode depicted in the capturedimage is decoded, and the first phone then employs the public key of thesecond phone to decrypt the decoded information. The recovered 3D poseinformation about the second smartphone is then compared withinformation about the pose of the first smartphone. If the two sets oforientation information agree, the second phone has been authenticatedto the first.

FIG. 4 shows another implementation. In this arrangement the firstsmartphone senses the barometric pressure. It then applies a hashingoperation to this pressure data to yield a key value. (So-called “fuzzy”or “robust” hashing is used, in which slight differences in the inputdata still yield the same hash value.)

The hash value is then applied as an encryption key to informationspecific to the first user, such as a user ID string (e.g., 10-digittelephone number, or email address, or Bluetooth name). The resultingstring is presented in textual form (ciphertext) on the display screenof the first phone.

The second phone, in turn, performs a parallel operation—querying itsown barometric sensor, and applying the same hashing operation to yieldthe same key value. Using its camera, the second phone then images thetext presented on the screen of the first phone, and applies an opticalcharacter recognition (OCR) operation to yield ASCII characters. Thistext string is decrypted using the just-computed hash as a decryptionkey. The decrypted text is then compared against its expected value(e.g., the first user's ID). The second phone then proceeds with atransaction involving the first smartphone, or not, based on the resultof this comparison.

FIG. 5 shows yet another implementation. This embodiment uses radiosignals sensed by the two phones as context data. The actual data usedmay be, e.g., the SSID of the strongest WiFi network detected by thephones, or the WiFi channel of the strongest network detected by thephones, or an identifier reported by a nearby RFID chip (aka near fieldchip, NFC), or other data received wirelessly by the phones, or derivedfrom such data.

As in the FIG. 4 arrangement, this context value is hashed by arobust/fuzzy hash operation. Instead of being used as a text cipher key,however, the output of this hash process is used as a spreading key fora spread spectrum image watermarking method. That is, the key defineswhere, or how, watermark payload data is to be encoded into hostimagery.

The host image can be arbitrary. In the detailed embodiment it is thedefault screen displayed by the first smartphone (e.g., an app display,or a desktop display). This imagery is digitally watermarked, using thespreading key, to convey an access credential. This access credential isinformation that can be used by a recipient to gain access toinformation relating to the first smartphone (or the first smartphoneuser).

For example, the encoded credential data may be a password that enablesaccess to a website or other online repository where information aboutthe first user (e.g., contact information) is stored. Or it may be a URLor other address identifier, indicating a location from whichinformation about the first user can be obtained (e.g., publishedcalendar data for the first user).

The second phone independently obtains corresponding radio signalinformation, and hashes it to yield a spreading key for a digitalwatermark process. The second phone captures imagery from the screen ofthe first phone, and then uses the key to attempt to decode the accesscredential information.

The second phone checks whether the watermark decoding operation wassuccessful (e.g., by CRC or other error checking test). If successful,the second phone uses the decoded access credential to obtaininformation. If the watermark was not successfully decoded, thisindicates an incorrect spreading key was used in decoding, indicatingthat authentication of the first smartphone failed.

(Although not shown in FIGS. 4 and 5, the detailed process can beextended to a two-way authentication process, e.g., following the modelof FIG. 3.)

Head-Mounted Display Systems, and Related Technology

As head-mounted display systems (HMDs) proliferate (such as the GoogleGlass product), new issues and opportunities arise.

A HMD may include one or more sensors, such as cameras, microphones, andaccelerometers. A HMD may also include sensors for detecting electricalor magnetic activity from or near the face and scalp (such as EEG andEMG, and myoelectric signals—sometimes termed Brain Computer Interfaces,or BCIs), etc. The system can also include other of the elements shownin FIG. 1. The display and some (or all) of the other system componentscan be provided as part of the head-worn apparatus. Alternatively, somecomponents can be provided elsewhere (e.g., a smartphone in the user'spocket, a computer in the cloud, etc.), and wirelessly linked to theheadworn apparatus. An illustrative HMD 60 (a Google Glass prototype),including a microphone 62, a camera 64 and a display 66, incommunication with a smartphone 68, is shown in FIG. 6. (Other HMDs aredetailed in references cited below.)

The camera(s) in the headworn apparatus may be outward facing, and/ororiented towards the wearer's eyes and face.

Although certain head-worn devices may include buttons foruser-interface purposes, it would be preferable not to use such buttonsfor most tasks.

One task is sharing information. Consider Alice and Bob, both wearingHMDs at work, and chatting at the water cooler. Alice mentions aninteresting journal article about self-driving cars she recently readonline. Bob asks Alice to send him a link to the article, and Aliceagrees.

As they were chatting, Alice's HMD detected that someone was speakingwith Alice. Alice's HMD identified her own speech by its strongamplitude and its spectral characteristics. (The HMD has an extensivehistory of sensing Alice's audio, so it is straightforward to recognizeher voice.) A self-directed camera on Alice's HMD may also provide videoimagery showing movement of her cheek and/or lips in synchrony with theaudio—providing an independent indication that certain of the sensedspeech is Alice's. (Indeed, the HMD system could discern the contents ofAlice's speech by lip-reading alone—irrespective of audio.)

The HMD system discerned that someone is in dialog with Alice by theback-and-forth nature of their audio. (Alice speaks, then a responsivesound is heard, etc.) From a library of familiar voice models, Alice'sHMD recognizes the other person as Bob.

In identifying Bob, Alice's HMD may rely, in part, on geolocation datato narrow the universe of possible speakers. Since Alice is at work,voice models for her co-workers (e.g., stored in the cloud) are checkedfor a match. Additionally or alternatively, a forward-facing camera maycapture imagery of Bob. Known facial recognition techniques serve toidentify Bob. (Again, the location context serves to narrow the field ofpossibilities.) The fact that the imagery shows that Bob's lips andfacial expressions are changing in synchrony with the other person'ssensed speech, helps confirm that Bob is the speaker with whom Alice isin dialog. Still further evidence of Bob's identity may be sensedwirelessly, e.g., by smartphone, Bluetooth, Zigbee, WiFi, NFC, etc.,emissions, which Alice's HMD system can receive and identify usingstored reference identification data (e.g., in a networked data store).

Having identified Bob, Alice's HMD presents a dossier of potentiallyhelpful information on a display for Alice to view. This includes areference image, profile information that Bob makes available to Alice,a synopsis of recent email (and spoken) communications between Alice andBob, calendar information, etc. Initially, none of this information isparticularly noteworthy, so Alice's HMD displays the data in a defaultfont (e.g., faint grey) —typically used for such background information.

During the course of their chat, Alice's HMD is understanding themeaning of the spoken dialog—possibly storing a transcription for a dayor a week (after which time it may be gracefully decayed). But it isprimarily listening for spoken contextual clues signaling informationthat may be useful to Alice.

The HMD searches data in an archive (local, and/or in the cloud) forhistorical information that might relate to the current discussion, sothat it can present same on Alice's display. When it hears the referenceto a recently-read article about “self-driving cars” it finds a cachedversion of an online article that Alice was reading yesterday—anddetermines that it is relevant to the dialog. The HMD displays acitation to the paper on Alice's display. Because the reference matchesthe description given by Alice, the HMD discerns that it is potentiallymaterial to the dialog, so renders the citation with a more prominentpresentation than the background information earlier mentioned, e.g., ina larger font, or a distinctive color or typeface. The display (whichserves as a virtual read/write whiteboard for Alice's visual use) mayhave a featured area for higher priority items—such as on a top of alist, or at the center of the display; the citation (and/or an image ofthe first page of the article) appears at such location.

By recognizing the speech of the dialog, and from semanticinterpretation of its meaning, Alice's HMD understands that Alice hasagreed to provide Bob a link to the paper. The HMD undertakes manyactions without obtaining express approval of Alice. However, where anaction involves communicating with another individual, rule data in astored set of rules dictates that the HMD not take such action unless aconfidence model indicates a 99+% certainty that the contemplated actionis one of which Alice would approve. In the present case, the calculatedconfidence falls short of this threshold (e.g., because Alice readseveral papers about self-driving cars ten days ago, and her referenceto reading a paper “recently” is ambiguous). Thus, her HMD seeks Alice'sconfirmation before sending Bob the link.

To solicit confirmation, Alice's HMD presents a text query on thedisplay screen, e.g., “Send the link to this article to Bob?” (The querycan optionally provide more information, such as the title of thearticle, or its URL.) Depending on the type of HMD display, this querymay appear—to Alice—to float in her field of view.

There are other ways to indicate to Alice that a response has beenrequested by the HMD. An aural prompt can made by the HMD, usingspeakers or other transducers near Alice's ear(s). The prompt mightsimply be an alerting beep tone, in conjunction with a text prompt. TheHMD may alternatively use speech to ask Alice the question “Send thelink to this article to Bob?”

Haptics are another way for the headset to call Alice's attention to aprompt.

Alice signals her assent, such as by letting her gaze linger on theHMD's displayed query for at least 1.3 seconds (a threshold interval oftime, set by Alice). This action can be sensed by a face-sensing cameraon her head-worn display (optionally with infrared illumination), inconjunction with eye tracking (e.g., corneal, pupil or retinal tracking)software.

Another approach for collecting Alice's response uses the same eyetracking mechanism, with two buttons floating in the field of view forher to focus on—Yes or No. Other feedback mechanisms can also be used.These include a double blink for a yes, or other facial movements thatcan be sensed by the HMD, such as a subtle facial move. These can becaptured with a camera on the HMD, or by electrical stimulus—includinguse of a Brain Computer Interface. (Commercially available BCIs includethe BrainGate from Cyberkinetics, the MindSet from Neurosky, and theEPOC neuroheadset from Emotiv.)

Another UI feedback paradigm—useful with binocular HMDs—relies onsensing the focal plane at which Alice's vision is focused (again by useof retinal, pupil or corneal tracking). If one visual prompt ispresented to apparently float at a great visual distance from Alice(e.g., infinity), and another appears to float just six inches in frontof her face, Alice's selection can be discerned from sensing at which ofthe two focal planes her eyes stereoscopically focus and dwell (or focusand then blink).

Such arrangement mitigates a problem with see-through HMDs: when inconversation with another person, a user's eyes may divert from lookingat the person, and instead look toward something of interest presentedin another part of the display. This betrays the user's distraction.Better is for certain visual prompts to be overlaid in the same visualdirection as the user's current gaze, but at infinity. The user can readsuch prompts by letting the eyes refocus at infinity—without changingtheir direction. This is less disconcerting to the person with whom theuser is speaking (if it is even noticed at all), and enhances continuityof the human-human dialog.

In one implementation, a “Yes” response is always presented to appear inone focal plane (e.g., infinity), and a “No” response always appear inthe other (e.g., near, such as 6 inches to 20 feet). In anotherimplementation, the response that is determined by the system to be mostlikely (e.g., based on historical experience, and the current context)is always in one focal plane (e.g., a near focal plane), and theresponse judged less probable is always at the second focal plane (e.g.,infinity).

(Other implementations can present more than two options, by indiciapresented at more than two apparent focal depths.)

Alice's HMD may use standardized user interface conventions by which shecan share, read, or print any document. Thus, even without a displayedquery asking whether a link to the document should be sent to Bob(prompted by recognition of Alice's spoken dialog with Bob), Alice cansummon a contextual menu by letting her gaze linger on a displayedobject (e.g., a link or document) for the threshold interval of time.This menu presents options suitable to the context, e.g., asking whethershe wants to read, print or share—posed as three displayed textualprompts. Again, Alice lets her gaze linger on a desired selection forthe threshold interval of time, to effect a choice. If a furtherselection is needed (e.g., share with who?), further text options arepresented, for user selection. In the sharing case, the list ofrecipients is contextually ordered, e.g., with the person(s) with whomAlice is in dialog at the moment at the top of the list.

Alice's HMD can identify Bob's network address in various ways. One isby use of his recognized face or voice. A table, or other data structureon the company network, may store data associating faces/voices/etc.with their corresponding network addresses. Another approach is for Bob(and others) to broadcast their network addresses to others, such as byshort-range wireless (e.g., Zigbee, Bluetooth), or by a servicediscovery protocol, such as Multicast DNS, or Apple's Bonjour.

By such arrangements, Alice's HMD can send the requested link about theself-driving car article to Bob. But Bob may want assurance that thedata really came from Alice. Moreover, Bob's HMD may have a filteringsystem, e.g., to assure that spam, viruses, and other unsolicited dataisn't presented on Bob's display.

So instead of accepting data packets sent from Alice's HMD, Bob's HMDresponds to Alice's transmission by requesting verification. The twosystems negotiate to determine which verification technique to use(e.g., Alice's HMD sends data indicating the verification protocolsenabled on her system, and Bob's HMD responds by selecting one that isenabled on his system). The selected verification protocol may be basedon shared context. Any of the arrangements detailed elsewhere in thisspecification can be used. An exemplary one is based on the audioenvironment. Alice's HMD processes ambient audio and sends resultingdata to Bob's HMD. Bob's HMD does likewise. Each system checks the datareceived from the other, to confirm expected correspondence with thecommon audio environment.

Once the two systems have authenticated themselves in this manner,Alice's HMD sends the requested article link to Bob's HMD. Known secureprotocols, such as HTTPS and digital signature technology, can be used.Desirably, however, no such action is taken until each device is certainthat it is communicating with another device in a common-contextenvironment.

As suggested, establishing trust will be a primary concern with HMDs.Throughout their dialog, Alice's HMD can be analyzing expressions onBob's face, modeling the expressions and the sensed audio to evaluatethe statistical probability that the expressions appear consistent withthe audio speech sensed by the microphone in Alice's HMD. (Alice wantsto guard against being spoofed by a recording of Bob's voice, with whichthe speaker's mimed expressions do not accurately correspond.)

To further aid in establishing trust, Bob's headworn apparatus maysample his EEG (or other biometric), and broadcast or otherwise providecorresponding data to Alice's HMD (e.g., via Bonjour, HTTPs, etc.).Electrical impulses associated with movement of Bob's mouth and otherfacial muscles are reflected in the EEG data sent to Alice. Alice canuse this data as another cross-check against data independently sensedby Alice's HMD sensors (e.g., imagery depicting Bob's facial movements,and audio reflecting his audio speech). If Alice's HMD discerns adiscrepancy in correspondence between the received biometric signals,and data her HMD senses from Bob, then something is amiss, andcorresponding defensive action can be taken.

Similarly, Bob's HMD can broadcast or otherwise provide data based onimagery from a self-facing camera, evidencing his facial expression,movement of lips and facial muscles, affect, etc. Likewise with otherdata sensed from Bob by Bob's HMD (e.g., audio). These data, too, can beused by Alice's HMD to enhance trust.

In a particular scenario, Bob' HMD senses his own facial affect, andtransmits corresponding information to Alice's HMD. Separately, Alice'sHMD assesses Bob's facial affect from its own imagery. Her own affectassessment is compared with that received from Bob's HMD forconsistency. (Bob can similarly authenticate Alice, in reciprocalfashion.)

Another issue with HMDs is their use by children, and parentalmoderation of such use.

One approach is to mediate the reality presented by HMDs. Through objectrecognition, visual overlays can obscure items that parents do not wishtheir children to encounter—in the physical or virtual world. (Adults,too, may choose to use such technology.)

Consider racy signage featuring a Calvin Klein model in Times Square.Imagery captured from the sign is machine-recognized/classified andfound to depict a male figure, with legs and chest bare. A HMD throughwhich this scene is viewed can augment the captured imagery with anoverlay that obscures (or clothes) the model's body.

In addition to simply recognizing (classifying) objects in real orvirtual imagery, such technology can also recognize actions, such as useof guns, or kissing. These, too, may be edited-out of imagery presentedto the HMD user.

Different social groups, with different sensitivities, can use HMDs thatare configured to mediate content in different ways. One line of HMDsmay be designed to conform to Sharia law—shielding users from viewingexposed skin and public depictions of kissing. Another line may bebranded by Disney and visually adapt imagery to—essentially—a “G”rating, omitting or obscuring items/actions unsuitable for smallchildren. Different organizations (Disney, the Islamic Sharia Council,etc.) may endorse different lines of HMDs, with different types ofinterventions.

While described in the context of visual imagery, audio can also beedited to reflect values or preferences of the listeners. (Some HMDs areconfigured so that no visual stimulus reaches the viewer except throughthe HMD system. HMDs do not typically process sound stimulus in such acomprehensive manner. However, it would be straightforward to do so,with earphones or earpieces that block sounds except those delivered bythe system.)

In both visual- and audio-content mediation, there may be some delayentailed in recognizing the content, and taking any necessary action. Inthe case of HMDs that perform both visual- and audio-content mediation,the audio is typically the controlling factor in the delay.

For example, audio mediation that replaces or omits curse words needs towait until an utterance is completed before concluding that it isobjectionable. In such systems, presentation of the video on the HMDdisplay is delayed by an interval of time greater than required forvisual mediation—to allow adequate time for audio mediation, andmaintain synchrony between the mediated audio and video.

In other cases, such a delay is not required. In television, forexample, the HMD can identify the displayed content through use ofwatermark of fingerprint techniques. Knowing the identity of thecontent, the HMD can query an online database, where the broadcaster orcontent producer has made available data identifying excerpts ofpossible concern. (The television station may populate such database inreal-time, e.g., by audio recognition of curse words, from a feed of thecontent ahead of the usual seven second broadcast delay.) By sucharrangement, the HMD can essentially look into the future and anticipateaction that will be required, allowing editing in real-time, as theaudio or visual content is rendered.

Content delivered by so-called “Over the Top” streaming models, such asNetflix and YouTube, can be handled similarly. Again, the HMD canidentify the content, and check an online repository detailing upcomingelements of the content. Corresponding action (e.g., obscuring ormodifying the content) can then be taken.

As an example, consider a family sitting down to watch a movie, whichthe parents believe—based on the coarse rating system provided by theMPAA—that that the PG content is going to be acceptable for their 8 and12 year old children. As the movie starts, however, several short scenesoccur that cause the parents to realize that the movie isn't appropriatefor their 8 year-old. Instead of turning the movie off, or sending theyoungster out of the room, other options exist. These include obscuringor modifying the visual or aural content for that child's HMD alone.These modifications can include dynamically editing the time-scale ofthe content. For audio, using known pitch invariant scaling techniques,and ambient noise suppression, the HMD can effectively skip a brief 0.5second utterance, and stretch the audio before and after the utteranceto seamlessly remove it for the child. Similar techniques are suitablefor use with imagery, such as using image segmentation, compressedsensing, etc. (An example of such image-altering technology is theContent Aware Scaling feature in Adobe Photoshop software.)

While described in the context of HMDs, it will be recognized that suchtechnology can also mediate other content delivery systems, such astelevisions and music players.

By such arrangements, different consumers experience different paralleluniverses. A gluten-intolerant consumer viewing a menu signboard atMcDonalds sees it rendered in the HMD display without items containinggluten. A lactose-intolerant consumer viewing the same menu sees itdifferently—omitting items containing lactose. A subscriber to theWeightWatcher HMD service sees still a different selection—showing onlyitems acceptable to the WeightWatcher's diet plan. TheWeightWatchers-configured HMD also prominently displays theWeightWatcher point count for each remaining item, as conspicuousgraphical overlays.

In each of these cases, the different consumers' HMDs all capturedimagery from the same, unabridged, menu signboard. But each altered therendered signage differently. (The original signboard descriptions wereall recognized by the HMDs using OCR; corresponding nutritionalinformation for each was then determined from an online database; andimagery presented to the consumer was altered accordingly.)

Different consumers can thus configure their own content mediationsbased on social affiliations and personal preferences; reality isexperienced with a user-selected editorial voice.

Users may also decide to switch mediation modes/personas to experiencethe world in different ways, at different times (e.g., akin to “PrivateBrowsing” modalities found in many web browsers, which some usersperiodically employ). In the private browsing construct, no history,state, etc. are stored.

Increasingly, consumers will be presented with electronic augmentationsvia HMDs. A QR code on a sign at a bus stop may cause a user's HMD tosuperimpose an advertising video in that region of the user's field ofview. Shazam-like recognition of a song may cause a user's HMD todisplay a corresponding music video in a corner of the display. Theseresponses are programmed by others (typically marketers), and mayconflict with user desires or values.

Again, such electronic augmentations are identified by known contentrecognition/classification technologies, and are edited/blocked inaccordance with user preferences. If a seven year old child curiouslyinteracts with a QR code found on a telephone pole, and the interactionresults in imagery of a Victoria's Secret model being transmitted to thechild's HMD for presentation in a corner of the display, the child'sSesame Street-branded HMD can respond by instead presenting a tutorialvideo at that display location, reminding the child that 6+6 is 12, and7+7 is 14.

Rather than mediate content after it is received, another aspect of thepresent technology concerns alerting users of content that may beencountered in particular environments.

Just as motion picture ratings “G,” “PG,” “PG-13,” etc., alert viewersas to content of movies, so too may environments have ratings. These cantake the form of machine-sensible data (e.g., visible indicia, audio orwireless signals) at certain venues, signaling the type of content thatmay be encountered in that venue.

Consider a store in a mall. A QR code at the entrance to the store, or asubliminal noise-like audio track, may signal information to HMDs andother devices about the particular content encountered inside. Is thereviolence? Allusion to drug use? Graphical display of sensuality? Foullanguage? Depictions of smoking? Crude humor? Etc? A parent mayconfigure a teen's HMD to be alert to such descriptors, and providetextual or auditory instruction to the child (e.g., “Don't go inthere”). If the child enters the space, the HMD may be configured tosend an email, text message, or other communication to the parentadvising of the conduct, or simply log the action in a data store forlater parental review.

The environment may also transmit to the HMD fingerprint data, or othercontent recognition/classification aids, by which the HMD might moreeasily discern different types of content in the venue (e.g., here is afile containing robust visual features by which an image of a personsmoking a cigarette, displayed inside the store, can be recognized).Such information can be cached in the HMD, and checked ahead of otherdata in trying to identify content by image fingerprinting. Being alertto the particular content hazards, the HMD can more quickly andaccurately respond by blocking or editing such content in theinformation rendered by the HMD to the teen.

Just as machine-sensible signals can alert a HMD about content that maybe encountered, such signals can also instruct a HMD about actions thatshould be taken (or not taken).

Consider facial recognition. This operation may be a backgroundoperation universally performed by most HMDs. Facial recognitionalgorithms are becoming ever-more accurate, and sources of referencefacial imagery from which such algorithms can make matches are becomingmore widespread. Some people, however, may object to such omnipresentdistributed surveillance, and want to opt-out.

Instead of wearing a ski mask or other disguise, such a person (Carol)can wear a visible badge, or her HMD or other device can emit acousticor radio signals, signaling to other HMDs that she does not wish to berecorded or recognized. Compliant HMDs worn by others (e.g., Dave)detect such signal and respond accordingly, e.g., by disabling datacollection (imagery and audio) from Carol, and disabling associatedfacial recognition and speech recognition.

Commonly, the face of the opting-out person can be located, within acamera's field of view, by reference to the detected origin of theopt-out signal. If the signal is a badge worn by Carol, imagesegmentation techniques (e.g., region-growing) can quickly locate herface. This corresponding region in the field of view of the image sensorin Dave's HMD can be disregarded—its data not stored (other thanancillary to tracking Carol's face as it may move in the image field).Or imagery corresponding to the facial region may be blurred to make itunrecognizable. Thus, no facial recognition can occur. If the opt-outsignal is acoustic or wireless, beam-forming techniques can be used(e.g., employing multiple microphones/antennas) to generally localizethe source of the signal. The face closest to this source is determinedto be the one that should be omitted/obscured from recorded data.

Similarly with speech. Opting-out-Carol is again located, relative toDave's HMD, by a technique such as image segmentation or audio/wirelessbeam-based localization. A sensing microphone array in Dave's HMD canthen be configured (e.g., by phasing) to place a null in Carol'sdirection, so that audio from Carol is relatively attenuated.Alternatively, imagery from Carol can be analyzed for purposes ofsensing her facial movements corresponding to speech. Audio sensed byDave's HMD microphone is analyzed for temporal correlation with thesesensed facial movements, and audio that corresponds to the facialmovements is attenuated to prevent its recognition. In still anotherarrangement, whenever opting-out-Carol is sensed to be speaking (e.g.,by reference to sensed facial movements), the microphone of Dave's HMDis momentarily disabled.

Carol can be identified in other ways, too. One example is byidentifying wireless emissions—such as by a telephone device. OrBluetooth, or NFC. Etc. Once identified, other information becomesmeaningful, like the location at which she was identified. Again, Dave'sHMD is respectful of her privacy desires. If he establishes a Bluetoothconnection with Carol, or senses other distinguishing attributepermitting her identification, receipt of the Carol's opt-out signalcauses his HMD not to log certain details of the transaction—such asdata identifying the geolocation at which the encounter took place.

It will be recognized that the foregoing arrangements have philosophicalkinship with the robot.txt standard, by which web sites can signal toGoogle, and other search engines that crawl the web, that they don'twant their content indexed and made available for search.

Another aspect of the technology concerns aiding Alice's hearing of Bob,e.g., in a noisy environment. Having segmented Bob's speech from thebackground noise (e.g., using multiple microphones on the front bezel ofthe HMD, or the three microphones on the Apple iPhone 5 device), inconjunction with beamforming, Alice's HMD can augment Bob's voice andsuppress the background noise by subtly amplifying his voice throughspeakers near Alice's ears. The same speakers can simultaneously play anoise suppression signal tailored to the environment (as is familiarfrom noise cancelling headsets—such as by Bose and others).

Still another aspect of the present technology is to enhancecross-cultural communication.

In some cultures, saying “no,” and otherwise expressing disagreement, isawkward or taboo. Thus, many cross-cultural dialogs are made difficult,because one party may be looking for a yes/no answer, and the otherparty—thinking “no,” won't say “yes” and is bound—out of respect—not tosay “no.” Facial expressions and tone of voice can provide clues tocommunications that sometime aren't explicitly expressed. Systems inwhich a camera senses facial expressions, and/or in which a camerasenses voice tone, can be trained (as detailed elsewhere) to discernemotion or meaning that is being expressed by a person, but just not inliteral words.

Relatedly, a user's own HMD can adapt its operation in accordance withthe user's sensed facial affect/mood, or other biometrically-sensedsignal. For example, if Bob is discerned—by his HMD—to be focused orconcerned (e.g., concentrating while writing software, or engaged in anintense dialog with his wife), his HMD may modify its behavior.

Some modifications concern how the HMD interacts with Bob. For example,a chime that normally sounds when email arrives, may be disabled or madequieter. Social network updates that occasionally appear on Bob'sdisplay may be repositioned more towards the periphery of his field ofview, or rendered in duller colors and/or with smaller font. Text newsfeeds that Bob hasn't paid much time with, historically, may besuppressed.

Other modifications concern how the HMD interacts with others. Bob's HMDmay become a bit more isolationist in the digital realm. It may stopbroadcasting Bob's IP address, and suspend other non-essentialtransmissions to others (e.g., by zeroconf or Multicast DNS). Moreover,it may reject inbound inquiries from other systems for information, anddecline invitations to participate in collaborative transactions withnearby devices.

Proactive queries that Bob's HMD usually sends to Google, allowing theHMD to present to Bob information relevant to presently-sensed audio,imagery, or other context, may similarly be suspended, or scaled-back innumber, or filtered so that only particularly salient queries are sent.

Another aspect of the present technology concerns sensing asymmetricpersonal relationships, and acting accordingly. This often involves useof social network data.

Consider two people who are about to engage in a transaction—perhapsakin to Alice and Bob's encounter. This time, however, they are notpeers. Instead, they are Professor and Student. Professor has a highersocial rank than Student, and this is reflected in online data sources.For example, such information can be harvested indicating that Professorreceived his PhD in Esoteric Biology from Harvard in 1972, and has beena Regents Professor in the Biology Department of State University since1986. In contrast, similar data about Student indicates that he is anunemployed undergraduate student in the Biology Department of StateUniversity.

Social network graph data from Facebook and LinkedIn shows thatProfessor has strong links (e.g., many interactions) with other academicsuperstars, whereas Student's strongest ties are to underachievers whowork at the neighborhood video store. This graph data may further showthat Student has mentioned Professor in several Facebook posts, butProfessor has never similarly mentioned Student. Moreover, the graphdata may show that Student sent two messages to Professor, asking for adraft of a journal article the Professor is writing, but that Professornever replied to Student.

Public records data may show that Professor lives in Upscale Acres, on a5 acre parcel having a tax valuation of a million dollars. An onlinerecords search to learn about Student's residence only shows that hiscredit history was checked six months ago by LowRentApartments, Inc.

From such information, an automated system concludes there is a statusdisparity between Professor and Student. This disparity can influencetheir electronic interaction.

In particular, consider Student encountering Professor on a Universitysidewalk. Each is wearing a HMD. Student asks Professor for a copy ofthe draft journal article. An electronic transaction process ensues—akinto that earlier detailed between Alice and Bob. However, due to theirdisparity, the HMDs of Professor and Student negotiate a differentauthentication protocol. In particular, Professor's HMD insists ondraconian terms from Student, requiring extraordinary effort by Student(or Student's HMD) to comply. Normally, Student's HMD would spurn sucharrogance and refuse. But having assessed and understood their disparatestatus relationships from the online data, Student's HMD decidescompliance is prudent, and performs all the rituals required byProfessor's HMD.

The required rituals may include, e.g., multiple familiar authenticationtechniques, such as repeated challenge-response tests, two-factorauthentication, etc. Professor's HMD may also require that Student's HMDsubmit something of value (including information) to escrow. The escrowterms may entitle Professor to claim or otherwise have rights (e.g.,redistribution rights) to the escrow deposit if Student fails to fulfillcertain requirements of their transaction (such as Student's violationof an agreement not to redistribute the draft journal article).

Student's HMD is set to normally require fairly strict authenticationbefore engaging in transactions outside family and close friends. Butgiven the present context, Student's HMD requests a lesser degree ofauthentication of Professor. Professor's HMD declines Student's modestauthentication request. Rather than abandon the transaction, Student'sHMD further relaxes its security requirements, deciding to continue thetransaction without requiring any authentication effort by Professor'sHMD—based on the information gleaned from the online sources. Student'sonly comfort is that data received from Professor will be subjected tovirus checking before it is accessed.

In like fashion, any two systems—as proxies for their respectiveusers—can adapt their modes of interaction in accordance with statusinformation gleaned from social networks and other online datarepositories. If one system is found to be relatively subservient to theother (e.g., in a social hierarchy), a disparity in standards may apply.Thus, digital devices understand the social context of users, and behaveaccordingly.

(Online services dedicated to ranking people's social status—so-calledsocial media analytic services—are growing in popularity. Klout<dot>comis one. It analyzes social network data to measure a person's socialinfluence, and identify the particular topical domains in which a personhas influence.)

Other Remarks

The presently-detailed technology is well suited for use in conjunctionwith other of the applicant's mobile-, context-, and socialnetwork-based technologies. These are detailed, e.g., in patentpublications 20110212717, 20110161076, 20100119208, 20100205628,20100228632, 20110159921, and 20120134548, and in pending applicationSer. No. 13/571,049, filed Aug. 9, 2012, Ser. No. 13/572,873, filed Aug.13, 2012, and Ser. No. 13/607,095, filed Sep. 7, 2012.

Having described and illustrated the principles of my work withreference to a few examples, it should be recognized that the technologyis not so limited.

For example, while described in the context of authenticating onesmartphone (or HMD) to another, the technology has a great variety ofother uses.

Consider, for example, social network “friending,” such as on Facebookor Linkedln. A user may elect to automatically friend other individualswith similar interests and/or demographics, who share aspects of currentor historical context.

A particular example is a user who attends a lecture on a specializedtopic (e.g., the technological implications of calligraphy), at auniversity auditorium. The user may decide that anyone who attends sucha specialized lecture, and is within a certain age range (or meets otherdemographic qualifier(s), such as shared membership in one or moreorganizations—as expressed by public profile data) would be aninteresting social network friend. The user launches an app that allowsdemographic parameters of acceptable friends to be input, and thenbroadcasts friend requests to others in the auditorium. (The requestsmay be sent across the university network using the Bonjour protocol(Apple's implementation of Zeroconf—a service discovery protocol), ordistributed using short range Bluetooth wireless, or by ad hoc wirelessnetworking, or by NFC, etc.) Interested responders qualify themselves toaccept the friend request, in part, by establishing their attendance atthe same lecture, such as by computing an audio fingerprint ofjust-captured audio, and sending it back to the user. The user's appchecks the history of recently-cached audio to confirm that the receivedfingerprint corresponds to audio that the user's smartphone also heard.After a responder has qualified himself/herself (at least in part) bybeing in the same acoustic context, the user's smartphone confirms theresponder as a friend. (The exchanged information naturally conveysother information needed to establish the social network relationship,e.g., each user's respective name on the social network, etc.)

Such an app can similarly be configured to automatically accept friendrequests issued by other individuals who are attending the lecture(again, optionally subject to specified demographic qualifiers).

Another use of the present technology is in context-fencing.

By way of background, geo-fencing refers to technology by which a useronly receives certain messages or information when at certaingeographies. For example, a user may write a shopping list that appearson the smartphone screen only when the user is near a grocery store.

In the broader context sense, a user may be presented with certaininformation only when certain context conditions are met. For example, auser may receive certain messages only when the user is with aparticular person (e.g., identified by shared auditory or other context,or Bluetooth device pairing). Conversely, the system may be configuredto not present certain messages when the user is with a particularperson. Similarly, if the user is at a concert, and a certain song isplayed, this auditory context may cause the smartphone to delivercertain messages, or activate otherwise dormant features. Likewise ifthe user is viewing a particular video program: the presence of acertain audio track may cause the smartphone to enable one or morecertain operations (e.g., message delivery) that are not otherwiseenabled.

A great number of context scenarios can influence when or how messagesare handled. Consider, in particular, scenarios in which watermark orfingerprint technology is used to identify ambient audio content.

One such arrangement provides ringback tones dependent on whattelevision show the user is watching. In particular, the user'ssmartphone can set the ringback tone (i.e., the audible indication heardthrough the telephone by a caller while the phone they are calling isringing) to the theme (intro song) of the show the user iswatching—thereby clueing-in the caller about the user's context, evenbefore the user answers the phone (or doesn't answer).

Another such arrangement sends an automated message (SMS/MMS) to a phonecaller, stating that the user is busy at an AC/DC concert, and will callback soon. (As part of the message, the user might specify that relatedcontent also be included, e.g., concert poster, etc.)

Wearable computing arrangements can make telephone interruptionschallenging, since the user doesn't have a physical control panelin-hand. This can be addressed, in part, by allowing the user's wearablecomputer to arbitrate who should be given attention—the person with whomthe user is physically speaking, or a person seeking to establishelectronic communication (e.g., by phone). In particular, the system cananalyze the social status of the competing parties. (Their identitiescan be discerned by voice recognition methods, and other techniquesdescribed herein, and by caller ID methods.) If the person physicallypresent with the user is found to be higher in rank, the system canrespond to the electronic communication with a message reporting thatthe user is talking to someone right now, but will respond to theirinquiry shortly. If the person physically present is determined to belower in rank, the system may allow the phone to ring, or otherwise passthe communication to the user. (The user may specify exceptions tonormal social status rankings, for example, giving the user's children aranking higher than online sources might indicate.)

Many context-based actions can take place in automobiles. Varioussensors can sense context, and stored rules can take action based oncontextual circumstances. Consider a father who specifies that hisnew-driver teen daughter may not drive with friends in the car. Thecar's computer system may have a user interface allowing the father tospecify a rule: if I am not present in the car, listen to the audio inthe passenger compartment. Identify (where possible) the names of thespeakers, and their number. Send me text message giving the names andnumber, if more than one voice is detected.

(The father's presence in the car is evidenced by Bluetooth signals fromhis smartphone, which the car has already been programmed to recognize.Attenuation of road noise, and sounds from the car radio, is a familiaraudio signal processing operation. Voices can be recognized by storedvoice models. If voice models are available, the number of differentspeakers can be identified by reference to how many models were matched.If voice models are not available, analysis of the audio (in conjunctionwith beamforming use of multiple microphones, where available) candistinguish several different voices.)

Still another aspect of the present technology is an automated pairingof devices, such as Bluetooth devices. The two devices can infer, fromshared context, that pairing is permitted. For example, the Bluetoothpairing protocol may follow stored rule data that causes devices tobroadcast their GPS, accelerometer, and ambient audio-based information,for other devices to receive. If two devices find that their GPS,accelerometer, and ambient audio data match, the stored rules mayautomatically associate the two devices. By such arrangement, if theuser drives a friend's car, or a rental car, the user's headset pairswith the vehicle's Bluetooth system without any user involvement. (Therules may specify that pairing not occur until certain conditions aremet, such as both accelerometers simultaneously reporting accelerationabove a threshold value.)

In some implementations, after the common-context conditions aredetected, the user may be prompted to confirm that pairing is desired.

Once paired, the user might receive messages that are based on contextgenerated by the car. These messages may be, e.g., service reminders,driving behavior advice for teens (based on car sensors and rules set byparents), likely road condition information (icy), reminders the usersent themselves about needing to obtain an item for the car (windshieldwipers), etc.

While the earlier-detailed arrangements employed a first smartphone'scamera to capture imagery from a second smartphone's screen, otherembodiments can convey information otherwise, such as by audio, radio,etc. For example, a first smartphone can emit an audio signal, using itsspeaker, that is a function of first phone context. The secondsmartphone can capture this signal with its microphone, and use its ownassessment of context to check the received information. Similarly, thefirst smartphone can issue a radio signal (e.g., Bluetooth, WiFi,Zigbee, RFID, etc.) that is a function of first phone context, which thesecond phone can then receive and check against its own contextualinformation.

Location (e.g., expressed by GPS coordinates) is a widely available typeof context data that can be used in the arrangements detailed herein.

It will be recognized that the present technology is well suited for usein challenge-response arrangements. In such arrangements, a firstsmartphone poses a challenge (question), and the second smartphone mustrespond with a valid answer.

The challenge question may be a straightforward inquiry about a contextvalue, e.g., report your location. The second phone provides therequested information, which is then evaluated by the first phone. Orthe challenge may involve information other than contextual data. Forexample, the first phone may provide a number, and instruct the secondphone to multiply that number by the fractional part of the secondphone's latitude, and return the result. Or the first phone may ask thesecond phone to add the current POSIX time (i.e., seconds elapsed sinceJan. 1, 1970 UTC) to the second phone's 10-digit telephone number, andreturn the result. Again, the first phone receives the response from thesecond phone, and compares it against the first phone's independentcalculation of the requested information, and proceeds with atransaction only if the two data agree within expected error tolerances.

The detailed types of context are only a few of the many that might beused. One particular type of context involves historical information,e.g., stored in a smartphone memory. Common history that is experiencedand recorded by two devices is a rich source of challenge/response data.Consider audio. Each smartphone may buffer recent ambient audio sampledby its microphone, and store it in association with correspondingtimestamp data (from the cell phone network-synchronized smartphoneclock). The first smartphone may challenge the second smartphone toidentify the highest-amplitude frequency component (or second-highest,etc.) in a 100 ms window centered at POSIX time 1318445064. Conventionmay establish, or the challenge can specify, that the frequencycomponent should be identified as a bin number in a 32 bin decompositionof the audio spectrum between 0 Hz and 3200 Hz. The second phoneperforms the requested calculation and returns the result to the firstsmartphone, which calculates the requested information independently,and compares the two answers.

Audio context may also be expressed by fingerprint data derived from theaudio.

While the foregoing discussion often focused on action of one device ina transaction (e.g., Alice's HMD), it will be understood that the otherdevice(s) involved in the transaction (e.g., Bob's HMD) performscorresponding operations.

Reference was made to analyzing different data streams for consistency(e.g., imagery depicting Bob's facial expressions, and audio recordinghis speech). One technique relies on machine learning. Facial imagery,audio and EEG data are collected from test subjects while they speak.The facial imagery is processed to identify robust feature points whosemovements can be tracked (e.g., using generic image processingtechniques such as SIFT or SURF, or by using domain specific featurepoints such as pupil location, nose, corners of mouth etc.). Likewise,distinctive features are identified in the audio and EEG data. Aheuristic classifier examines the processed data to discern features intwo or more data streams that appear associated, e.g., particular EEGsignal features that recur each time a subject's mouth opens, or eyesblink. Confidence in the associations increases the more frequently theyco-occur. With a few hours of training across several differentsubjects, a satisfactory data set comprising sensor signals that areconsistently correlated is developed. This stored data then serves as areference model against which live data sensed from speakers (e.g., Bob)is judged. These models can be used by the HMD locally or in the cloudfor multiple purposes. Alice's HMD can do a confidence calculation thatit is indeed Bob speaking (i.e., his voice is being created by theobserved face).

(Related heuristic techniques have been employed in development ofmachine lip-reading systems—correlating lip movements to audio speech.That work is applicable here. Sample references appear below.)

A HMD with a self-directed camera was noted above. One such camera maycapture imagery that includes one or both the user's eyes. Another maycapture imagery that includes at least part of the user's cheek(s)and/or lips. Still another may capture all such information. Any suchfacial information can be useful in the detailed arrangements.

While reference has been made to smartphones and HMDs, it will berecognized that this technology finds utility with all manner ofdevices—usually portable, but also fixed. Portable music players,desktop computers, laptop computers, tablet computers, set-top boxes,televisions, netbooks, other wearable computers, servers, etc., can allmake use of the principles detailed herein.

Particularly contemplated smartphones include the Apple iPhone 5, andsmartphones following Google's Android specification (e.g., the VerizonDroid Eris phone, manufactured by HTC Corp., and the Motorola Droid 4phone).

The term “smartphone” (or “cell phone”) should be construed to encompassall such devices, even those that are not strictly-speaking cellular,nor telephones.

(Details of the iPhone, including its touch interface, are provided inApple's published patent application 20080174570.)

The design of smartphones and other computers referenced in thisdisclosure is familiar to the artisan. As reviewed above, each includesone or more processors, one or more memories (e.g. RAM), storage (e.g.,a disk or flash memory), a user interface (which may include, e.g., akeypad, a TFT LCD or OLED display screen, touch or other gesturesensors, a camera or other optical sensor, a compass sensor, a 3Dmagnetometer, a 3-axis accelerometer, a 3-axis gyroscope, one or moremicrophones, etc., together with software instructions for providing agraphical user interface), interconnections between these elements(e.g., buses), and an interface for communicating with other devices(which may be wireless, such as GSM, CDMA, W-CDMA, CDMA2000, TDMA,EV-DO, HSDPA, WiFi, WiMax, or Bluetooth, and/or wired, such as throughan Ethernet local area network, a T-1 internet connection, etc.).

The processes and arrangements detailed in this specification may beimplemented as instructions for computing devices, including generalpurpose processor instructions for a variety of programmable processors,including microprocessors (e.g., the Atom and A5), graphics processingunits (GPUs, such as the nVidia Tegra APX 2600), and digital signalprocessors (e.g., the Texas Instruments TMS320 series devices), etc.These instructions may be implemented as software, firmware, etc. Theseinstructions can also be implemented in various forms of processorcircuitry, including programmable logic devices, field programmable gatearrays, field programmable object arrays, and application specificcircuits—including digital, analog and mixed analog/digital circuitry.Execution of the instructions can be distributed among processors and/ormade parallel across processors within a device or across a network ofdevices. Processing of data may also be distributed among differentprocessor and memory devices. “Cloud” computing resources can be used aswell. References to “processors,” “modules” or “components” should beunderstood to refer to functionality, rather than requiring a particularform of implementation.

Software instructions for implementing the detailed functionality can beauthored by artisans without undue experimentation from the descriptionsprovided herein, e.g., written in C, C++, Visual Basic, Java, Python,Tcl, Perl, Scheme, Ruby, etc. Smartphones and other devices according tocertain implementations of the present technology can include softwaremodules for performing the different functions and acts.

Known browser software, communications software, and media processingsoftware can be adapted for many of the uses detailed herein.

Although features and arrangements are described, in some cases,individually, applicant intends that they also be used together.Conversely, while certain systems are detailed as including multiplefeatures, applicant conceives that—in other embodiments—the individualfeatures thereof are usable independently.

Similarly, while this disclosure has detailed particular ordering ofacts and particular combinations of elements, it will be recognized thatother contemplated methods may re-order acts (possibly omitting some andadding others), and other contemplated combinations may omit someelements and add others, etc.

Likewise, aspects of the different embodiments can readily be changedand substituted. (E.g., embodiments described as conveying informationby watermarking can instead convey information by text presentation orbarcode. Data used as a cipher key in one embodiment can be used as awatermark spreading key in another. Etc.)

Although disclosed as complete systems, sub-combinations of the detailedarrangements are also separately contemplated.

While certain aspects of the technology have been described by referenceto illustrative methods, it will be recognized that apparatus configuredto perform the acts of such methods are also contemplated as part ofapplicant's inventive work. Likewise, other aspects have been describedby reference to illustrative apparatus, and the methodology performed bysuch apparatus is likewise within the scope of the present technology.Still further, tangible computer readable media containing instructionsfor configuring a processor or other programmable system to perform suchmethods is also expressly contemplated.

References to communication between two devices (including references totransmitting and receiving) should be understood to encompasscommunications through intermediary devices and systems. (E.g., Alice'sHMD may not communicate directly with Bob's HMD, but rather throughintermediate routers, computers, etc.)

This specification refers, in some instances, to recent history (e.g.,audio recently sampled). The meaning of recent can vary with differentapplications. Typically, it means within the past hour, and morecommonly within the past 10, or 3 or 1 minutes, or 30 or 10 seconds.However, in some applications this period can go back a day, a week, ayear, or longer.

Likewise, the error tolerance within which two values are deemed tocorrespond (or not) depends on application. In some applications,correspondence within 5% may be used. In others a looser standard may beemployed (e.g., 10% or 15% or more). In others a tighter tolerance maybe applied (e.g., agreement within 3%, or 1%, or 0.3%). Etc.

Exemplary digital watermarking techniques are taught in the assignee'spatent documents U.S. Pat. No. 6,590,996 and 20100150434. Robust hashingof audio is described, e.g., in patent documents 20020178410 and U.S.Pat. No. 6,996,273. Robust generation of shared key data (e.g., forBluetooth pairing) is detailed in Kirovski, The Martini Synch: JointFuzzy Hashing Via Error Correction, Security and Privacy in Ad-hoc andSensor Networks, LNCS Vol. 4572, pp. 16-30 (2007).

Examples of audio fingerprinting are detailed in patent publications20070250716, 20070174059 and 20080300011 (Digimarc), 20080276265,20070274537 and 20050232411 (Nielsen), 20070124756 (Google), U.S. Pat.No. 7,516,074 (Auditude), and U.S. Pat. Nos. 6,990,453 and 7,359,889(Shazam).

Electroencephalography sensors and associated systems are detailed,e.g., in published patent documents 20090112077, 20090281408,20090156925, 20080177197, 20090214060, 20120245450, and 20110040202 (allto NeuroSky), 20070185697 (Microsoft), and 20080235164 (Nokia).

Lip reading systems are detailed, e.g., in published patent documents20110071830 (Hyundai), 20100332229 (Sony), 20100189305 (Eldon), and20100079573 (Sony Ericsson). See, also, Cetingul et al, DiscriminativeAnalysis of Lip Motion Features for Speaker Identification andSpeech-Reading, IEEE Trans. on Image Processing, Vol. 15, No. 10, 2006,pp. 2879-2891; and Perez, et al, Lip Reading for Robust SpeechRecognition on Embedded Devices, Int. Conf. Acoustics, Speech and SignalProcessing (ICASSP), 2005, pp. 473-476, and papers cited therein.

Head-mounted display systems, and related technology, are detailed,e.g., in published patent documents U.S. Pat. Nos. 8,235,529, 8,223,088,8,203,605, 8,183,997, 8,217,856, 8,190,749 and 8,184,070 (Google);20080088936, 20080088529, 20080088937 and 20100079356 (Apple); and20120229909, 20120113092, 20050027515 and 20120068913 (Microsoft).

Patent publication US20110251493 details technology by whichphysiological parameters for a person can be discerned by analysis offacial imagery.

Arrangements for recognizing pornography from imagery are detailed,e.g., in published patent documents 20090123064, 20080159627, and20080159624.

Social network graphs, their uses, and related technology, are detailedin applicant's pending application Ser. No. 13/425,339, filed Mar. 20,2012, and Ser. No. 13/572,873 (Appendix A), filed Aug. 13, 2012.

Prior art uses of context in security applications are detailed in Sigg,Context-Based Security: State of The Art, Open Research Topics and aCase Study, 5th ACM Intl Workshop on Context-Awareness for Self-ManagingSystems, 2011; and in Mayrhofer, The Candidate Key Protocol ForGenerating Secret Shared Keys From Similar Sensor Data Streams, Securityand Privacy in Ad-hoc and Sensor Networks, pp. 1-15, 2007 (attached asAppendices A and B to the provisional parent application 61/546,494).

Bluetooth is familiar to artisans. A Wikipedia article about Bluetoothis attached as Appendix D to parent provisional application 61/546,494.

The prior art references identified herein are assumed to be familiar toartisans implementing the present technology.

To review, certain aspects of the technology involve a device obtainingone or more context parameters, such as date and/or time of day,information based on visual stimulus recently sensed by the firstdevice, sensed radio transmissions, sensed RFID data, orientation sensordata, magnetometer data, acceleration sensor data, audio classificationdata output from a logical sensor, activity classification data, orbarometric sensor data. The device generates a signal based, at least inpart, on one or more of these parameters, and transmits the signal to asecond device. The second device may use the signal to authenticate atransaction, establish an ad hoc network with the first device,establish a logical link with the first device, make a determination,etc.

The transmission of the signal to the second device can be by an imagepresented by the first device, and sensed by a camera in the seconddevice. The image can convey a machine-readable representation of thesignal, such as a barcode, or a digital watermark steganographicallyencoded into some other image display. Alternatively, the signal can betransmitted by audio, radio, or otherwise.

Other aspects of the technology involve shared context determinations inconnection with social networking. For example, a friend request may beissued, or accepted, based at least in part on a finding that twodevices share certain contextual similarities.

Still other aspects of the technology involve triggering presentation ofa message to a user, at a time when certain context conditions are met(as determined by reference to obtained context data).

Yet other aspects employ archived historical context information isarchived in connection with other devices, e.g., in challenge-responseauthentication.

Still other aspects concern headworn apparatuses that sense biometricinformation from the head of a user (e.g., EEG, facial imagery), andtransmit corresponding data for use by another device (e.g., inconnection with an authentication transaction).

While this specification earlier noted its relation to the assignee'sprevious patent filings, it bears repeating. These disclosures should beread in concert and construed as a whole. Applicant intends that thefeatures, methods, elements, concepts and enablement details disclosedin the present application be combined with the features, methods,elements, concepts and enablement details described in those relatedapplications. Implementation of such combinations is straightforward tothe artisan from the provided teachings.

To provide a comprehensive disclosure, while complying with the 35 USCSection 112 mandate of conciseness, applicant incorporates-by-referencethe patent and other documents referenced herein. Such materials areincorporated in their entireties, even if cited above in connection withspecific of their teachings. These references disclose technologies andteachings that applicant intends be incorporated into the arrangementsdetailed herein, and into which the technologies and teachings detailedherein be incorporated.

In view of the wide variety of embodiments to which the principles andfeatures discussed above can be applied, it should be apparent that thedetailed embodiments are illustrative only, and should not be taken aslimiting the scope of the invention. Rather, I claim as my invention allsuch modifications as may come within the scope and spirit of thefollowing claims and equivalents thereof.

I claim:
 1. A method employing a first device, comprising the acts:using one or more sensors in said first device obtaining one or morecontext parameters selected from the list: information based on visualstimulus recently sensed by the first device; sensed radiotransmissions; sensed RFID data; orientation sensor data; magnetometerdata; audio classification data, received from a logical sensor fed byinformation both from an audio sensor and another sensor in said firstdevice; activity classification data; and barometric sensor data;generating an image based, at least in part, on said obtainedparameter(s); and presenting the image, for sensing by a camera of asecond device, to authenticate a transaction, in which the generatingact comprises encrypting a first of said obtained context parameterswith a private key associated (a) with the first device, or (b) with auser of the first device, and representing the encrypted first contextparameter as a barcode or digital watermark in the image.
 2. The methodof claim 1 that includes generating the image based, at least in part,on a parameter obtained from a logical sensor, the logical sensor fed byinformation from plural different sensors in said first device.
 3. Themethod of claim 1 in which the generating includes generating said imagebased, at least in part, on two or more of said obtained parameters. 4.The method of claim 1 in which the generating includes generating saidimage based, at least in part, on three or more of said obtainedparameters.
 5. The method of claim 1 in which the generating includesencoding information based on said obtained parameter(s) in image form.6. The method of claim 1 in which the generating includes digitallywatermarking imagery to convey information based on said obtainedparameter(s).
 7. The method of claim 1 in which the generating comprisesgenerating an image based, at least in part, on said audioclassification data.
 8. The method of claim 1 in which generating animage includes encoding first data in said image based on the obtainedparameter.
 9. The method of claim 8 that further includes encryptingsaid first data in accordance with a private key associated with thefirst device.
 10. The method of claim 1 that further includes sensingthe presented image using the camera of the second device, andauthenticating the transaction through use of information conveyed bythe sensed image.
 11. The method of claim 1 that further includes: witha camera of the first device, sensing a barcode or watermark imagepresented by the second device, said image encoding an encrypted payloadbased on a context parameter from said list that was sensed by thesecond device; with a hardware processor in the first device, decryptingsaid payload using a public key associated (a) with the second device,or (b) with a user of the second device, to thereby obtain the contextparameter from said list that was sensed by the second device;determining if said obtained context parameter that was sensed by thesecond device matches said context parameter obtained using the firstdevice; and taking an action based on a result of said determining. 12.An article of manufacture including a non-transitory computer-readablemedium having instructions stored thereon that, if executed by a firstdevice, cause the first device to perform operations comprising:obtaining, from one or more sensors in the first device one or morecontext parameters selected from the list: sensed radio transmissions;sensed RFID data; orientation sensor data; magnetometer data; audioclassification data, received from a logical sensor fed by informationboth from an audio sensor and another sensor in said first device;activity classification data; and barometric sensor data; generating animage based, at least in part, on said obtained parameter(s); andpresenting the image, for sensing by a camera of a second device, toauthenticate a transaction, in which said generating operation comprisesencrypting a first of said obtained context parameters with a privatekey associated (a) with the first device, or (b) with a user of thefirst device, and representing the encrypted first context parameter asa barcode or digital watermark in the image.
 13. The article of claim 12in which said computer-readable medium stores further instructions that,if executed by the first device, cause the first device to performoperations comprising: with a camera of the first device, sensing abarcode or watermark image presented by the second device, said imageencoding an encrypted payload based on a context parameter from saidlist that was sensed by the second device; decrypting said payload usinga public key associated (a) with the second device, or (b) with a userof the second device, to thereby obtain the context parameter from saidlist that was sensed by the second device; determining if said obtainedcontext parameter that was sensed by the second device matches saidcontext parameter obtained using the first device; and taking an actionbased on a result of said determining.
 14. A first device including adisplay, a processor, one or more sensors, and a memory, the memorycontaining instructions that configure the device to perform a methodthat includes: obtaining from said sensors one or more contextparameters selected from the list: (a) sensed radio transmissions; (b)sensed RFID data; (c) orientation sensor data; (d) magnetometer data;(e) audio classification data, received from a logical sensor fed byinformation both from an audio sensor and another sensor in said firstdevice; (f) activity classification data; and (g) barometric sensordata; generating an image based, at least in part, on said obtainedparameter(s); and presenting the image, for sensing by a camera of asecond device, to authenticate a transaction, in which said generatingoperation comprises encrypting a first of said obtained contextparameters with a private key associated (a) with the first device, or(b) with a user of the first device, and representing the encryptedfirst context parameter as a barcode or digital watermark in the image.15. The device of claim 14 in which said generating comprises generatingan image based, at least in part, on context parameter (a) from saidlist.
 16. The device of claim 15 in which said generating comprisesgenerating an image based also on one or more of context parameters(b)-(g) from said list.
 17. The device of claim 15 in which saidgenerating comprises generating an image based also on two or more ofcontext parameters (b)-(g) from said list.
 18. The device of claim 14 inwhich said generating comprises generating an image based, at least inpart, on context parameter (b) from said list.
 19. The device of claim18 in which said generating comprises generating an image based also onone or more of context parameters (a), (c), (d), (e), (f) or (g) fromsaid list.
 20. The device of claim 18 in which said generating comprisesgenerating an image based also on two or more of context parameters (a),(c), (d), (e), (f) or (g) from said list.
 21. The device of claim 14 inwhich said generating comprises generating an image based, at least inpart, on context parameter (e) from said list.
 22. The device of claim21 in which said generating comprises generating an image based also onone or more of context parameters (a), (b), (c), (d), (f), or (g) fromsaid list.
 23. The device of claim 21 in which said generating comprisesgenerating an image based also on two or more of context parameters (a),(b), (c), (d), (f), or (g) from said list.
 24. The device of claim 14 inwhich said generating comprises generating an image based, at least inpart, on context parameter (f) from said list.
 25. The device of claim24 in which said generating comprises generating an image based also onone or more of context parameters (a), (b), (c), (d), (e), or (g) fromsaid list.
 26. The device of claim 24 in which said generating comprisesgenerating an image based also on two or more of context parameters (a),(b), (c), (d), (e), or (g) from said list.
 27. The device of claim 14 inwhich said instructions configure the device to perform further actsincluding: with a camera of the first device, sensing a barcode orwatermark image presented by the second device, said image encoding anencrypted payload based on a context parameter from said list that wassensed by the second device; decrypting said payload using a public keyassociated (a) with the second device, or (b) with a user of the seconddevice, to thereby obtain the context parameter from said list that wassensed by the second device; determining if said obtained contextparameter that was sensed by the second device matches said contextparameter obtained using the first device; and taking an action based ona result of said determining.